A business or enterprise may store information about various items in the form of electronic records. For example, a company might have an employee database where each row in the database represents a record containing information about a particular employee (e.g., the employee's name, date of hire, and salary). Moreover, in some cases large amounts of data may need to be retrieved from a data source (e.g., when copying information to a back-up data store). In some cases, a system may simultaneously retrieve and/or process records from the data source using multiple processing threads. Note that two threads may be prevented from accessing a single record 210 at the same time (e.g., to avoid errors or conflicts).
The processing threads may be assigned groups or ranges of records to be retrieved. For example, range 1 might be defined as including data records associated with part numbers “0” through “1,000,” range 2 might be defined as including data records associated with part numbers “1,001” through “2,000,” etc. Each processing threads may then be assigned to one of ranges and begin to retrieve and/or process the records from that range. When a processing thread has finished retrieving all of the records in its assigned range, it may simply halt or wait.
Note, however, some of the records may be empty or less difficult to process as compared to other records 310. As a result, a processing thread might finish retrieving all of the records it's range before other threads have finished retrieving their records (e.g., because one range had a relatively large number of empty records). This can result in a thread waiting (not performing useful work) and reduce the efficiency of the system.
Accordingly, a method and mechanism for efficiently retrieve data from a data source using multiple processing threads may be provided in accordance with some embodiments described herein.